In the video based in-air signature verification system, the existed methods cannot meet the requirement of accuracy, real time, robustness for fingertip tracking. To solve this problem, the Tracking-Learning-Detection (TLD) method based on temporal context was proposed. Based on the original TLD algorithm, the temporal context massage, namely the prior knowledge that the movement of fingertip is continuity in two adjacent frames, was introduced to narrow the search range of detection and tracking adaptively, thereby improving tracking speed. The experimental results on 12 public and 1 self-made video sequences show that the improved TLD algorithm can accurately track fingers, and tracking speed can reach 43 frames per secend. Compared with the original TLD tracking algorithm, the accuracy was increased by 15% and the tracking speed was increased more than 100%, which make the proposed method meet the real-time requirements for fingertip tracking.